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ABSTRACT 

A  common  problem  in  the  design  of  expert  systems  is  the 
definition  of  rules  from  data  obtained  in  system  operation  or 
simulation.  While  it  is  relatively  easy  to  coUect  data  and  to  log 
the  comments  of  human  operators  engaged  in  experiments, 
generalizing  such  infonnadon  to  a  set  of  rules  has  not  previously 
been  a  straightforward  task.  This  paper  presents  a  statistical 
method  for  generating  rule  bases  from  numerical  data,  modvated 
by  an  example  based  on  aircraft  navigadon  with  multiple 
sensors.  The  specific  objecdve  is  to  design  an  expert  system 
that  selects  a  satisfacto^  suite  of  measurements  from  a 
dissimilar,  redundant  set,  given  an  arbitrary  navigadon  geometry 
and  possible  sensor  failures.  This  paper  describes  the 
systematic  development  of  a  Navigadon  Sensor  Management 
(NSM)  Expen  System  from  Kalman  Filter  covariance  data.  The 
development  method  invokes  two  stadstical  techniques:  Analysis 
of  Variance  (ANOVA)  and  the  ID3  algorithm.  She  ANOVA 
technique  indicates  whether  variations  of  problem  parameters 
give  statistically  different  covariance  results,  and  the  ID3 
algorithm  identifies  the  relationships  between  the  problem 
parameters  using  probabilistic  knowledge  extracted  from  a 
simulation  example  set.  ANOVA  results  show  that  statistically 
different  position  accuracies  are  obtained  when  different 
navigation  aids  are  used,  the  number  of  navigation  aids  is 
changed,  the  trajectory  is  varied,  or  the  performance  history  is 
altered.  By  indicating  that  these  four  factors  significantly  affect 
the  decision  metric,  an  appropriate  parameter  framework  was 
designed,  and  a  simulation  example  base  was  created.  The 
example  base  contained  over  900  training  examples  from  nearly 
300  simulations.  The  ID3  algorithm  then  was  applied  to  the 
example  base,  yielding  classification  "rules"  in  the  form  of 
decision  trees.  The  NSM  expert  system  consists  of  seventeen 
decision  trees  that  predict  the  performance  of  a  specified 
integrated  navigation  sensor  configuration.  The  performance  of 
these  decision  trees  was  assessed  on  two  arbitrary  trajectories, 
and  the  performance  results  are  presented  using  a  predictive 
metric.  The  test  trajectories  used  to  evaluate  the  system's 
performance  show  that  the  NSM  Expen  adapts  to  new  situations 
and  provides  reasonable  estimates  of  sensor  configuration 
performance. 
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ABSTRACT 

A  common  problem  in  the  design  of  expert  systems  is  the 
definition  of  rules  from  data  obtained  in  system  operation  or 
simulation.  While  it  is  relatively  easy  to  collect  data  and  to  log 
the  comments  of  human  operators  engaged  in  experiments, 
generalizing  such  information  to  a  set  of  rules  has  not  previously 
been  a  straightforward  task.  This  paper  presents  a  statistical 
method  for  generating  rule  bases  from  numerical  data,  motivated 
by  an  example  based  on  aircraft  navigation  with  multiple 
sensors.  The  specific  objective  is  to  design  an  expert  system 
that  selects  a  satisfactory  suite  of  measurements  from  a 
dissimilar,  redundant  set,  given  an  arbitrary  navigation  geometry 
and  possible  sensor  failures.  This  paper  describes  the 
systematic  development  of  a  Navigation  Sensor  Management 
(NS.Mi  Lxpen  System  from  Kalman  Filter  covariance  data.  The 
development  method  invokes  two  statistical  techniques;  Analysis 
of  Variance  (ANOVA)  and  the  ID3  algorithm.  The  ANOVA 
technique  indicates  whether  variations  of  problem  parameters 
give  statistically  different  covariance  results,  and  the  1D3 
algorithm  identifies  the  relationships  between  the  problem 
parameters  using  probabilistic  knowledge  extracted  from  a 
simulation  example  set.  ANOVA  results  show  that  statistically 
different  position  accuracies  are  obtained  when  different 
navigation  aids  are  used,  the  number  of  navigation  aids  is 
changed,  the  trajectory  is  varied,  or  the  performance  history  is 
altered.  By  indicating  that  these  four  factors  significantly  affect 
the  decision  metric,  an  appropriate  parameter  framework  was 
designed,  and  a  simulation  example  base  was  created.  The 
example  base  contained  over  900  training  examples  from  nearly 
300  simulations.  The  ID3  algorithm  then  was  applied  to  the 
example  base,  yielding  classification  "rules"  in  ^e  form  of 
decision  trees.  The  NSM  expert  system  consists  of  seventeen 
decision  trees  that  predict  the  performance  of  a  specified 
integrated  navigation  sensor  configuration.  The  performance  of 
these  decision  trees  was  assessed  on  two  arbitrary  trajectories, 
and  the  performance  results  are  presented  using  a  predictive 
metric.  The  test  trajectories  used  to  evaluate  the  system's 
performance  show  that  the  NSM  Expert  adapts  to  new  situations 
and  provides  reasonable  estimates  of  sensor  configuration 
performance. 

INTRODUCTION 

Knowledge  acquisition  is  a  major  problem  in  the 
development  of  rule-based  systems.  The  tools  developed  to  date 
are  not  designed  to  extract  information  from  data  for  which  no 
generalizations  are  known  a  priori.  Instead,  these  tools  either 
rely  on  the  expert  to  provide  examples  from  which  rules  are 
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generated  or  try  to  capture  the  expert's  problem-solving 
methodology  with  interviewing  techniques  Jl].  Unfortunately, 
it  often  is  difficult  for  experts  to  describe  their  problem-solving 
methods  or  to  detail  the  factors  that  come  into  play  during  the 
resolution  of  a  problem.  It  is  exactly  this  type  of  knowledge  that 
is  needed  to  design  rule-based  systems. 

Since  the  early  1970's  adaptive  navigation  has  been  viewed 
as  a  highly  desirable  candidate  for  development  in  next- 
generation  aircraft  [2].  It  is  envisioned  that  future  aircraft  will 
have  multi-sensor  capability  for  navigation  tasks  requiring  high 
reliability,  optimal  performance,  and  increased  automation. 
With  multi-sensor  capability,  the  task  of  sensor  configuration 
selection  and  management  will  become  an  additional  pilot 
burden. 

The  performance  of  multi-sensor  navigation  systems  (more 
commonly  known  as  "integrated"  or  "hybrid"  systems)  has  been 
explored  since  the  late  I960’s  when  results  from  modem  control 
theory  provided  techniques  for  sensor  mixing  and  optimal  state 
estimation  [3].  Hybrid  systems  refer  to  externally  referenced 
navigation  systems  that  "aid"  an  on-board  inertial  navigation 
system  (INS)  using  an  optimal  state  estimation  mechanization. 
Hvbrid  navigation  systems  combine  the  high-  and  low- 
frequency  accuracy  propenies  of  INSs  and  external  navigation 
aids  (navaids)  respectively.  Many  radio  navigation  and  on¬ 
board  systems  aiding  INS  have  been  modelled  and  their 
performance  covariance  results  obtained  (4-8).  When  radio 
navigation  systems  are  only  partially  operational,  results  show 
that  improv^  navigation  performance  is  obtained  over  that  of 
the  pure  INS  [4].  Therefore  it  becomes  advantageous  to  keep 
panially  operational  systems  as  candidates  for  integrated  sensor 
mixing  purposes. 

With  a  large  number  of  available  navaids,  choosing  an 
optimal  or  near-optimal  sensor  set  becomes  a  large  combinatorial 
problem.  Convergence  towards  an  optimal  sensor  configuration 
requires  an  exhaustive  computer  search  utilizing  simulation 
results  as  the  basis  for  selection.  In  contrast,  a  small  number  of 
available  navaids  reduces  the  decision  space  considerably. 
Hence,  a  dilemma  occurs;  increasing  sensw  capability  (and  thus 
reliability  and  performance)  increases  decision-making 
complexity. 

The  selection  of  an  optimal  configuration  requires  the 
application  of  some  decision  criteria.  Most  often,  designers 
choose  between  navaids  based  on  the  relative  accuracies  of  each 
system  using  a  hierarchical  approach  [9],  This  approach  is 
"knowledge-based"  in  the  sense  Aat  the  nominal  perform^ce  of 
the  systems  is  well-known  and  that  this  knowledge  is  built  into 
the  sensor  hierarchy.  The  current  hierarchical  designs  are  ttot  as 
"robust"  with  respect  to  sensor  availability  and  performance 
changes  as  is  necessary  for  future  sensor  management  systems 
(10].  Instead,  these  hierarchies  represent  "rules-of-thumb"  that 
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are  useful  in  only  the  simplest  cases.  They  do  not  resolve 
sensor  configuration  problems  when  more  detailed  information 
must  be  considered  -  for  example  when  the  number  of  each 
available  navaid  is  specified,  when  panially  operational  systems 
remain  viable  candidates,  and  when  trajectory  effects  degrade 
system  performance.  It  becomes  necessary  to  explore  factors 
other  than  the  performance  of  nominally  operating  navaids  to 
determine  how  these  factors  affect  the  decision-maidng  process, 
and  to  exploit  the  potential  of  hybrid  systems. 

The  statistical  technique  Analysis  of  Variance  (ANOVA) 
[11]  was  used  to  identify  the  factors  that  cause  variation  in 
navigation  performance.  Once  the  important  factors  were 
identified,  the  relationships  between  them  were  determined.  The 
1D3  algorithm  [12,13],  an  inductive  inference  technque  based  on 
the  probabilistic  occurrence  of  events,  was  used  to  find  these 
attribute  relationships. 

The  development  of  a  navigation  sensor  management  expen 
system  using  the  ANOVA/ID3  technique  [14]  is  described  in 
this  paper.  The  NSM  system  controls  the  selection  of  multi¬ 
sensor  configurations.  The  methodology  is  applicable  to  any 
problem  where  the  development  of  knowledge  bases  from  multi¬ 
factor  data  studies  is-desired. 

INTEGRATED  NAVIGATION  SYSTEMS 

Optimal  estimation  techniques  are  used  to  combine  inertial 
and  radio  navigational  systems  in  order  to  provide  stable 
continuous  inertial  navigation  information  [15].  The  errors 
exhibited  by  these  "hybrid"  systems  depend  on  the  accuracy  of 
the  aiding  system,  and  navaid  accuracies  are  functions  of  many 
factors  such  as  navaid  type,  number  of  similar  navaids,  and 
trajectory  parameters  such  as  distance  from  the  navaid  and 
whether  the  aircraft  is  approaching  or  receding  from  the  station. 
The  sensor  selection  criteria  depend  on  the  relative  importance  of 
these  factors.  Five  external  radio  navigation  and  two  on-board 
navaids  were  used  to  update  a  medium-accuracy  (10  N.  Mi/hr) 

I.NS.  Hybrid  system  performance  was  simulated  using  the 
linearized  inertial  navigation  error  model  and  navaid 
measurement  models  as  inputs  into  the  optimal  estimation  filter. 
The  hybrid  errors  were  updated  at  a  specified  navaid  fix  rate. 
The  systems  simulated  were  (1)  Global  Positioning  System 
(GPS),  (2)  Long-Range  Navigation  System  (LOIL\N),  (3) 
Tactical  Navigation  System  (TACAN),  (4)  Distance  Measuring 
Equipment  (DME),  (5)  VHF  Omnidirectional  Range  (VOR),  (6) 
Doppler  radar,  and  (7)  air  data  sensor.  The  operational  theory 
and  the  mathematical  models  used  to  simulate  the  navaids  and 
the  inertial  navigation  error  model  are  discussed  in  detail  in  [  14]. 

The  numerically-stable  discrete-time  U-D  implementation  of 
the  Kalman  Filter  equations  was  used  to  mix  the  inertial  system 
and  navaid  information  optimally,  providing  covariance 
estimates  of  the  navigation  errors  (e.g.,  north/east  position) 
[14,17].  Each  nonlinear  measurement  equation  was  linearized 
with  respect  to  the  inertial  navigation  states  to  obtain  the 
observation  matrix  used  in  the  U-D  measurement  update 
equations.  Since  sensor  errors  were  taken  into  consideration  in 
the  measurement  models,  the  inertial  error  stale  vector  was 
augmented  with  the  sensor  shaping  filter  dynamics  (e.g., 
random  bias,  first-order  Markov  model)  to  formulate  the  hybrid 
navigation  model.  Additionally,  the  measurement  noise  time 
history  was  simulated.  As  the  aircraft  moves  along  its  trajectory 
relative  to  ground-based  navaid  stations,  the  measurement  noise 
characteristics  change.  Therefore  an  equation  for  a  distance-  or 
time-v^ing  measurement  covariance  matrix  was  found  in  order 
to  realistically  model  ground-based  radio  navigation  systems. 
According  to  Ref.  17,  GPS  measurement  noise  increases  in  a 
similar  way;  as  the  satellite  descends  near  the  aircraft's  horizon, 
the  noise  increases.  To  simulate  time-varying  measurement 


noise  for  the  ground-  and  satellite-based  navigation  systems, 
each  noise  variance  was  modelled  as  the  sum  of  initial  and 
range-dependent  variances.  The  latter  component  increases 
linearly  with  the  square  of  the  distance  from  the  station  or 
satellite. 

Position  accuracy  was  selected  for  the  rule-based  system 
decision  metric.  Here,  position  accuracy  is  defined  as  the  root 
sum  of  squares  (RSS)  of  the  north  and  east  component  errors. 
The  RSS  decision  metric  provides  sufficiently  consistent 
quantities  to  compare  hybrid  performances.  For  a  detailed 
discussion  of  the  RSS  decision  metric,  the  reader  is  directed  to 
Ref  14. 

HYBRID  NAVIGATION  SIMULATION  RESULTS 

Using  the  RSS  position  error  metric  to  measure  hybrid 
system  performance,  the  following  U-D  filter  simulations  were 
performed: 

1.  Single-type  hybrids:  GPS,  LORAN,  TACAN,  DME, 

VOR,  Doppler  Radar,  or  Air  Data  Sensor  aiding  an  INS 

2 .  Number  of  stations  used  in  a  single-type  hybrid 

3.  Multi-tj-pe  hybrids:  Combinations  of  different  navaid  types 

aiding  an  INS 

4.  Aircraft  trajectories  simulated:  High-performance, 

commercial,  general  aviation 

Comparisons  of  Single-Type  Hybrid  Performance 

Consider  the  four  ground  stations  A,  B,  C,  and  D  spatially 
oriented  with  respect  to  the  high-performance,  commercial,  and 
general  aviation  trajectories  in  Fig.  1 .  The  four  ground  stations 
are  simulated  as  LORAN  slaves,  TACAN.  DME.  or  VOR 
stations.  Figure  2  shows  the  performance  differences  of 
ground-based,  GPS,  and  on-board  type  hybrid  systems.  When 
the  results  from  all  ground  station  A  types  (LORAN,  TACAN, 
DME,  VOR)  are  compared  on  the  high-performance  trajectory  , 
the  relative  performance  from  best  to  worst  may  be  listed  as 
follows:  (1)  LORAN,  (2)  TACAN.  (3)  DME,  and  (4)  VOR. 
ror  example,  a  hybrid  system  utilizing  LORAN  slave  station  A 
provides  better  performance  than  a  hybrid  system  utilizing 
TACAN  A;  a  TACAN  A  hybrid  in  turn  outperforms  a  DME  A 
hybrid  which  in  turn  outperiorms  a  VOR  A  hybrid.  This  pattern 
is  repeated  for  stations  B,  C,  and  D  [14].  The  best  hybrid 
performance  was  obtained  from  three  GPS  satellites  aiding  the 
INS.  Figure  2  also  shows  how  the  pierfoimanccs  of  the 
Doppler  radar  hybrid  and  the  air  data  sensor  hybrid  compare 
with  the  GPS  and  ground-based  navaid  hybrids. 

Referring  to  the  LORAN  results  in  Fig.  3,  there  is  a 
striking  variation  in  the  performance  of  the  individual  Stations 
A-D;  this  figure  reveals  that  single  stations  of  the  same  type 
aiding  an  INS  give  highly  variable  performance  results.  "The 
same  variability  in  performance  of  the  remaining  ground-based 
single-station  navaids  was  found  [14].  From  Fig.  3.  the 
variation  in  Station  A-D's  performances  is  attributed  to  the 
position  of  each  ground  station  relative  to  the  aircraft's 
trajectory.  For  example,  LORAN  Slave  A  gives  the  smallest 
position  error  of  the  four  stations;  referring  to  Fig.  1,  the 
aircraft  makes  a  close  approach  to  Slave  A  on  the  trajectory's 
second  leg.  Hence  the  RSS  error  becomes  very  small.  These 
errors  begin  to  increase  towards  the  end  of  the  trajectory  leg, 
due  to  the  increasingly  uncertain  north  component.  In  contrast, 
LORAN  Slaves  B,  C,  and  D  are  farther  from  the  aircraft's 
trajectoiy.  The  first  trajectory  leg  results  in  good  relative  north 
information  to  B,  C,  and  D,  whereas  the  cast  component 
uncertainty  grows  due  to  the  lack  of  relative  east  information. 
The  variations  in  performance  observed  from  Stations  A-D  are 
due  to  trajectory  effects;  using  Station  B  instead  of  A  to  update 
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Figure  1.  Aircraft  Trajectories  Used  in  Simulations 


the  INS  is  equivalent  to  using  A  and  changing  the  aircraft's 
irajeciorv'. 

Effect  of  Increasing  the  Number  of  Navaids  in  a 
Hybrid  System 

Next,  the  effect  of  the  number  of  ground  stations  was 
studied  by  simulating  all  possible  combinations  of  single, 
double,  and  triple  stations  formed  from  stations  A-D.  There  are 
six  possible  combinations  of  two  stations  and  four  combinations 
of  three  stations  that  may  be  integrated  to  aid  the  INS.  These 
simulations  were  earned  out  for  LORAN,  TACAN,  DME  and 
VOR. 

Referring  to  the  LORA.N  results  in  Fig.  4,  the  performance 
variation  among  the  double  station  combinations  and  triple 
station  combinations  is  less  pronounced  than  the  single  station 
variations.  The  magnitude  of  the  RSS  errors  decreases 
dramatically  when  two  stations  are  used  instead  of  one  station. 
The  RSS  errors  decrease  further  when  three  stations  are  used, 
although  the  magnitude  differences  are  not  as  great.  The  reason 
why  the  RSS  magnitudes  of  the  double-  and  triple-station 
combinations  are  much  lower  is  that  the  aircraft  receives  the  best 
navigation  information  available.  This  also  explains  why  there 
is  much  more  variation  in  the  results  for  the  double  station 
combinations  than  for  the  triple  stations.  Similar  performance 
trends  were  observed  for  GPS,  TACAN,  DME,  and  VOR  [14J. 

Effect  of  Trajectory  on  Hybrid  Performance 

It  already  has  been  shown  that  an  aircraft's  trajectory 
relative  to  a  single  ground  station  hybrid  plays  an  important  role 
in  the  estimator's  performance.  The  RSS  results  in  Fig.  5 
illustrate  the  performance  differences  of  the  LORAN  Slave  A 
hybrid  on  the  high-performance,  commercial  transpon,  and 
general  aviation  trajectories.  Two  parameters  that  contribute  to 
these  performance  differences  are  distance  to  a  station  and 
heading  with  respect  to  a  station.  A  third  trajectory  parameter 
that  contributes  to  a  hybrid  system's  performance  is  the  number 
of  heading  changes  along  the  trajectory.  The  effect  of  heading 
changes  is  discussed  in  more  detail  in  [14].  Trajectory  factors 
affect  the  INS  dynamics,  which  in  turn  affects  the  error 
estimation  performance.  The  trajectory  factors  also  change  the 
measurement  dynamics  since  the  measurements  are  dependent 
on  the  trajectory's  geometric  propenies  and  aircraft  states  (such 
as  velocity).  The  results  in  Fig.  5  clearly  show  that  when  the 
trajectory  changes,  the  navaid  selection  decision  most  likely 
changes  as  well  since  the  relative  accuracies  of  the  navaids 
change. 
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Figure  2.  Performance  of  Satellite,  Ground 

Station-Based  and  On-Board  Hybrid 
Navigation  Systems 


Hybrid  Performance  of  Mixed  Navaids 

Figure  6  shows  variou:;  combinations  of  integrated  navaids. 
The  individual  performances  of  LORAN  Slave  B,  Doppler 
radar,  and  Air  Data  hybrids  are  shown  in  Fig.  6  along  the  high- 
performance  trajectory.  The  LORAN/Dopplcr  and  LORAN/Air 
data  hybrids  also  are  ploned  in  this  figure  for  comparison.  Both 
combinations  gave  better  results  than  their  individual 
components  operating  alone.  For  example,  the 
LORAN/Doppler  combination  ouqicrformcd  the  LORAN  hybrid 
and  the  Doppler  hybrid;  similarly,  the  LORAN/Air  Data 
combination  gave  better  results  than  did  the  LORAN  alone  or  the 
Air  Data  sensors  alone.  The  latter  combination  did  slightly 
better  than  Doppler  hybrid  on  this  trajectory  after  the  initial 
transient  period.  These  results  show  that  good  navigation 
performance  is  still  obtainable  when  a  "failed"  LORAN  system 
(only  one  slave  station  operational)  is  integrated  with  an  on¬ 
board  navaid  such  as  Doppler  radar  or  a  standard  equipment  air 
data  sensor. 
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Figure  3.  Performance  of  Single-Station  LORAN 
Navaids  Aiding  an  INS. 


DEVELOPME.NT  OF  A  NAVIGATION  SENSOR 
MANAGEMENT  EXPERT  SYSTEM 

This  section  describes  a  novel  methodology  that  uses 
established  statistical  techniques  to  develop  the  NSM  expert 
from  the  simulation  data.  The  primary  function  of  this  expert 
system  is  to  select  the  external  navaid  sensors  that  provide  the 
smallest  possible  RSS  position  error  from  a  large  set  of  available 
sensors.  The  Analysis  of  Variance  (ANOVA)  technique  [1 1]  is 
used  to  identify  the  factors  that  make  statistically  significant 
contributions  to  the  decision  metric.  Then,  the  ID3  algorithm 
determines  the  relationships  between  these  factors  [11,13J. 

Identifying  Important  Factors  Using  ANOVA 

The  ANOVA  technique  was  applied  as  follows;  first,  the 
mean  value  of  the  RSS  position  error  and  the  variance  for  all  the 
simulations  were  computed.  The  ANOVA  model  decomposes 
the  variance  into  a  sum  of  variances,  each  associated  with  a 
potentially  contributing  factor.  Over  two  hundred  simulations 
w  ere  performed,  and  the  data  were  used  in  a  four-factor  navaid 
experiment.  The  goal  of  the  experiment  was  to  identify  which 
of  the  factors  (navaid  type,  number  of  ground  stations, 
trajectory  effects,  performance  history)  and  their  interactions  had 
statistically  significant  impacts  on  the  RSS  position  error.  The 
factor  states  used  in  the  ANOVA  experiment  were: 
Navaids={VOR,  DME,  LORAN,  TACAN,  GPS);  Number  of 
Ground  Stations={One,  Two,  Three);  Trajectory  Type=(High- 
Performance,  Commercial  Transport,  General  Aviation,  from 
Fig.  1);  Time  Interval  =  (1,  II,  Ill,  IV).  Since  each  trajectory 
consists  of  four,  fifteen-minute  legs,  the  "Time  Interval"  factor 
refers  to  the  RSS  performance  obtained  within  each  fifteen 
minute  time  frame.  Four  single-station,  six  double-station,  and 
four  triple-station  hybrids  were  simulated  using  combinations  of 
Stations  A-D  in  Fig.  1. 

The  ANOVA  results  [14]  show  that  three  of  the  four  factors 
are  strongly  significant  with  99%  confidence;  the  fourth  factor, 
trajectory,  was  shown  to  be  weakly  significant  (90% 
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Figure  4.  Performance  of  Double-Station  LOR.AN 
Navaids  Aiding  an  INS 


confidence).  The  latter  result  suggested  that  additional 
investigation  into  the  effect  of  trajectory  on  RSS  position  error  is 
necessary  for  more  specific  trends  to  be  observed.  Indeed  the 
term  "trajectory"  is  extremely  vague;  the  results  from  Scheffe 
comparison  tests  suggest  that  "trajectory"  should  be 
decomposed  into  attributes  that  describe,  in  better  detail,  what 
these  effects  really  are.  For  example,  some  trajectory  attributes 
include  distance  from  a  station,  airspeed,  and  whether  the 
aircraft  is  approaching  or  receding  from  the  station.  Scheffe 
multiple  comparison  tests  were  applied  to  the  navaid  and 
number  of  ground  station  factors  to  identify  the  specific 
differences  within  each  groups;  for  example,  the  RSS 
performance  difference  between  GPS  and  TACAN,  all  other 
factors  being  equal,  was  statistically  significant.  On  the  other 
hand,  the  RSS  performance  difference  between  LORAN  and 
TACAN  with  all  other  factors  being  equal,  was  not  statistically 
significant.  This  means  that  a  LORAN  hybrid  could  perform 
bener  or  worse  than  a  TACAN  hybrid,  depending  on  the  values 
of  the  other  factors  (e.g.,  number  of  ground  sutions).  The 
multiple  comparison  test  results  yielded  the  same  performance 
ranking  depicted  in  the  graphical  results  (e.g..  Fig.  2),  while 
utilizing  the  information  content  of  a  large  number  of 
independent  simulations.  Further  investigation  into  the  ANOVA 
interaction  effects  revealed  that  the  tanking  should  be  cautiously 
applied  to  single-station  hybrids,  since  these  are  highly-sensitivc 
to  trajectory  effects.  The  complete  factor  analysis  results  are 

given  in  Ref.  14.  In  summary,  the  ANOVA  and  Scheffe 
methods  systematically  identified  trends  in  the  simulation  data 
without  recourse  to  tedious  graphical  analysis. 

Extracting  Rules  Using  Induction:  The  ID3  Algorithm 

The  ID3  Algorithm  uses  inductive  inference  to  extract  rules 
[13]  from  a  training  set  of  examples.  The  problem  space  is 
described  in  terms  of  attributes,  where  each  attribute  is 
characterized  by  a  set  of  values  that  define  the  possible  "states." 
For  example,  in  the  previous  section,  the  navaid  type  and 
number  of  ground  stations  were  shown  to  be  attributes  anecting 
RSS  position  error.  The  attribute  values  for  the  factor  "navaid 


Figure  5.  Comparison  of  RSS  Results  for  LORAN 
Hybrids  on  Three  Different  Trajectories 


type'  uere  (GPS,  LORAN.  TACAN,  D.\1H,  VOR),  and  the 
aitribtite  values  for  tlie  factor  number  of  stations"  were  {One, 
Two,  lliree).  Hence  there  is  a  clear  connection  between 
ANO\  A  and  ID.^  problem  structures.  .NNOVA  factors  are  ID3 
attributes,  and  A.NOVA  factor  levels  are  ID3  attribute  values. 

An  imponant  problem  in  designing  an  inductive  inference 
algorithm  is  identifying  the  attributes  that  span  the  problem 
space  most  efficiently,  so  that  the  resulting  decision  tree  is  as 
compact  as  possible.  The  ID3  algorithm  selects  the  most 
iniDori.int  attributes  using  an  information-theoretic  measure 
(liM)  that  minimizes  the  number  of  tests  (attribute  nodes) 
necessiiry  to  classify  a  problem.  The  ID3  algorithm  utilizes  a 
spliiting  strategy  [12]  to  decide  which  attribute  provides  the 
most  information  from  the  example  set.  A  detailed  example 
illustrating  how  the  splitting  strategy  is  used  to  construct 
classification  rules  is  given  in  [14], 

Developing  fhe  ID3  Attribute  Framework  Using 
AN()\'A  Results 

Up  to  three  ground  stations  (four  GPS  satellites)  were 
included  as  possible  configurations.  Time-weighted 
measurement  effects  are  included  in  the  attribute  framework 
using  RSS  position  error  classification  codes  representing  the 
hybrid's  performance  on  a  preceding  trajectory  leg.  The 
trajectory  effects  were  separated  into  the  following  attributes; 
geodetic  distance  from  a  ground  station,  line-of-sight  angle  from 
the  station,  and  the  direction  of  flight  (approaching  or  receding) 
relative  to  a  ground  station.  TTie  distance  from  a  ground  station 
is  an  important  attribute  since  the  signal-to-noise  ratio  decreases 
as  the  distance  to  the  station  increases.  The  direction  of  flight 
with  respect  to  the  station  influences  position  accuracy  through 
its  effect  on  the  line-of-sight  angle.  The  trajeaory  panuneters 
were  computed  for  each  of  the  high-performance,  jet  transport, 
and  general  aviation  trajectories  on  each  trajectory  leg.  The 
maximum  and  minimum  distances  to  the  aiding  station  were  also 
determined  on  each  trajectory  leg,  in  addition  to  the  difference 
between  the  maximum  and  minimum  distances. 

When  more  than  one  station  was  used,  the  attributes  were 
redefined  slightly.  The  maximum  and  minimum  distances  then 


Figure  6.  Comparison  of  RSS  Results  with  LORAN, 

Doppler  Radar,  and  Air  Data  Sensor  Hybrid 
Combinations 


referred  to  the  closest  and  farthest  distances  computed  to  the 
stations.  The  distance  difference  is  the  algebraic  difference 
between  the  farthest  and  closest  distances  determined  on  the 
trajectory  leg.  A  similar  definition  was  applied  to  the  line-of- 
sight  angle;  from  the  angles  computed  to  each  station,  the 
]arge.st  and  smallest  were  selected.  The  1D3  algorithm's  task 
was  then  to  determine  how  these  attributes  were  related  to  each 
other  and  to  the  RSS  performance. 

The  classification  scheme  chosen  to  represent  the  RSS 
position  error  endnode  in  the  decision  trees  is  depicted  in  Table 
I.  Since  an  approximate  prediction  of  the  RSS  position  error 
was  of  interest,  it  was  appropriate  to  represent  the  RSS 
performance  in  terms  of  an  error  range. 

Table  I  RSS  Position  Error  Classification 
Scheme 


Accuracy 

{High]  [Medium]  [Low] 


Error 

Code 

Error 

Code 

Error 

Code 

(N.  Mi.) 

(N.  Mi.) 

(N.  Mi.) 

0.(X)-0.02 

c-1 

0.10-0.20 

c-6 

1.0- 1.5 

c-15 

0.02-0.04 

c-2 

0.20-0.30 

c-7 

1.5-2.0 

c-16 

0.04-0.06 

c-3 

0.30-0.40 

c-8 

2.0-2.5 

c-17 

0.06-0.08 

c-4 

0.40-0.50 

c-9 

2.5-3.0 

c-18 

0.08-0.10 

c-5 

0.50-0.60 

c-10 

3.0-3.5 

c-19 

0.60-0.70 

c-11 

3.5-4.0 

c-20 

0.70-0.80 

c-12 

4.0-4.5 

c-21 

0.80-0.90 

c-13 

4.5-5.0 

c-22 

0.90-1.00 

C-14 

>^.00 

c-23 

The  velocity,  distance,  and  line-of-sight  angles  were 
expressed  in  terms  of  ranges  instead  of  individual  values,  so  that 
the  expert  system  weights  trends  more  heavily  than  specific 
examples.  This  renders  the  expert  system  more  adaptable  to 
new  conditions,  because  matches  between  the  actual  and 
knowledge-base  cases  could  be  obtained  more  frequently. 


The  example  set  was  developed  using  the  attribute 
framework  described  above.  The  RSS  position  errors  for  each 
simulation  were  classified  on  each  trajectorv'  leg  using  the 
scheme  in  Table  I.  The  1D3  example  base  was  then  created 
from  each  single-,  double-,  and  triple-station  simulation. 

NSM  Decision  Trees 


decision  tree.  Figure  7  also  shows  that  distance,  ground 
velocity,  LOS  angle,  and  hybrid  performance  history  are 
significant  factors  that  enable  a  prediction  of  the  RSS  error  to  be 
made.  The  RSS  classification  results  verify  that  the  closer  the 
aircraft  is  to  a  station(s),  the  smaller  is  the  RSS  error;  other 
results  show  that  the  larger  is  the  LOS  angle,  the  smaller  is  the 
RSS  error  [14], 


The  NSM  example  set  was  divided  into  seventeen  smaller 
example  sets.  The  GPS  and  on-board  navaid  examples  were 
grouped  into  one  expert,  whereas  the  ground-based  navaid 
examples  were  divided  according  to  navaid  type  and  time  (15- 
minute  intervals).  The  1D3  algorithm  constructed  decision  trees 
for  each  of  the  seventeen  small  expert  systems  that  comprise  the 
hu'ger  ,NS.M  Expert.  The  breakdown  of  the  NSM  Expert  into 
smallei  systems  provides  greater  manageability  of  the  training 
example  base.  The  total  number  of  examples  used  to  develop 
the  NS.M  Expert  System  was  nine  hundred  and  thirty-two.  In 
total,  two  hundred  and  sixty  Kalman  Filter  covariance 
simulations  were  performed  to  formulate  the  complete  NSM 
example  set.  An  additional  thirty-seven  simulations  were 
pert'ormed  to  obtain  a  decision  tree  to  estimate  RSS  performance 
when  different  navaid  types  are  combined.  The  NSM  expert 
system  prompts  the  user  for  a  set  of  fltght  conditions 
commensurate  with  the  aitribute/value  lists  used  in  the  example 
set,  and  the  resulting  RSS  classification  code  is  returned  to  the 
user  from  the  decision  tree. 

•■3  ts'pical  decision  tree  obtained  for  the  ground-based 
navaids  is  exemplified  b\  the  TACAN  results.  Figure  7 
presents  the  decision  trees  for  single-,  double-,  and  triple-station 
combinations  on  the  first  fifteen-minute  trajectory  leg.  Here,  the 
majority  of  the  testing  nodes  are  trajectory  parameters  (distance, 
LOS  angle,  direction  of  flight  with  respect  to  the  station(s)). 
The  top  or  root  node  in  Fig.  7  is  the  aircraft's  direction  of  flight. 
This  IS  expected  because  the  distance  and  LOS  angle  attributes 
are  dependent  on  directional  motion.  Distance,  LOS  angle,  and 
groundspeed  are  results  of  the  aircraft's  motion,  and  hence, 
represent  more  specific  problem  parameters;  therefore  it  is 
expected  that  these  parameters  appear  at  a  lower  depth  in  the 


The  expected  performance  of  the  GPS  system  on  each 
trajectory  leg  is  shown  in  Fig.  8.  Note  that  the  aircraft's 
groundspeed  plays  an  important  role  in  the  GPS  hybrid  s 
performance.  Velocity  affects  the  measurement  dynamics 
(history)  and  is  therefore  classified  as  a  trajectory  effect.  From 
Fig.  8,  the  two-satellite  hybrids  are  more  sensitive  to  these 
velocity  effects  than  are  the  three-  and  four-satellite  hybnds. 

Finally,  the  decision  tree  showing  what  position  error  range 
is  expected  when  different  navaid  types  are  integrated  in  a 
hybrid  system  is  presented  in  Fig.  9.  Note  that  the  decision  tree 
is  not  specified  for  a  given  trajectory  leg.  The  RSS  position 
errors  for  these  simulations  were  averaged  over  the  entire  flight 
time  for  the  high-performance  trajectory.  The  tree  is  organcBd  in 
Emis  of  the  naxigtjion  method  ased  ( 1 )  Distance- Vekx±y  (p-V),  (2)  Beaxg- 
Velociiy  (9-V),  (3)  DLsnrice-Beiring  (p-6),  (4)  Distince-Distance  (p-p).  <5) 
Bearing-Bearing  (&-6),  and  (6)  Vefacitx-Vdoeiix'  (V-Vj,  These  lesuhs  show 
iha  LCRAN  is  a  better  distance-measuring  navaid  than  DME  and 
that  Doppler  Radar  is  a  better  velocity-measuring  system  than 

the  Air  Data  Sensor  when  p-V  navigation  is  used.  The  p-6 
results  show  that  it  is  possible  to  obtain  performance  when 
LORAN  and  VOR  are  used.  The  LORAN/DME  hybrid  gives 
better  results  than  two  DME  stations  but  worse  performance 
than  two  LOR.AN  stations.  By  far  the  worst  results  at 
obtained  using  two  VOR  stations.  As  discussed  before,  the 
VOR  system  is  the  least  accurate  measurement  device  of  the 
seven  systems  studied,  which  great!)  affects  INS-VOR  hybrid 
results. 


Figure  7.  Decision  Trees  Predicting  RSS  Position  Error  Range  for  an  INS 
Aided  by  TACAN  During  the  First  IS  Minutes  of  Flight 


Figure  8  Decision  Tree  Predicling  RSS  Performance 
for  an  INS  Aided  by  GPS 


Figure  9  Decision  Tree  Predicling  RSS  Performance  When 

Different  Navaid  Combinations  are  Used  to  Aid  an  INS 


PERFORMANCE  RESULTS  OF  NSM  EXPERT 
SYSTEM 

It  is  imponant  to  quantify  the  NSM  Expert's  performance 
for  several  test  scenarios,  in  terms  of  how  well  it  predicts  a 
given  hybrid's  RSS  position  error.  It  is  also  important  to  gain 
insight  into  the  factors  that  affect  the  system's  performance,  so 
that  these  factors  can  be  exploited  in  future  system  development. 


Two  high-performance  trajectories  were  used  in  the 
performance  evaluation  of  the  NSM  Expert.  The  two 
trajectories  each  consist  of  four  fifteen-minute  legs.  Trajectory 
#2's  flight  pattern  was  in  a  counter-clockwise  direction,  whereas 
clockwise  flight  patterns  were  used  to  develop  the  NSM  Expert 
(Fig.  1).  Additionally,  the  takeoff  point  on  'Trajectory  #2  was 
five  degrees  farther  north  than  the  training  trajectories’  takeoff 
points.  These  trajectory  differences  change  the  measurement 
and  INS  dynamics,  and  hence  the  hybrid  performance. 
Trajectory  #2  was  designed  this  way  intentionally,  so  that  the 
NSM  Expert  System's  adaptability  could  be  determined. 

Single-,  double-,  and  triple-station  combination  hybrids 
were  simulated  on  each  test  trajectory  for  each  of  the  DME, 
■VOR,  TACAN,  and  LORAN  systems.  The  combinations  were 
formed  using  fbur  ground  stations  located  as  in  Fig.  1  with 
respect  to  each  other.  Additionally,  two-,  three-,  and  four- 
satellite  hybrids  were  simulated  on  the  test  trajectories,  as  were 


Doppler  Radar  and  Air  Data  sensor  hybrids.  In  total,  sixty 
covariance  simulations  were  performed  for  the  two  test 
trajectories. 

Test  Trajectory  Data  Preparation,  Performance 
Metrics,  and  Results 

The  performance  results  for  each  of  the  sixty  simulations 
were  classified  on  each  trajectory  leg  according  to  the  scheme  in 
Table  1.  The  total  number  of  matches  was  counted  on  each  leg 
of  each  test  trajectory  for  the  seven  navaid  types  studied.  A 
match  was  declared  between  the  actual  and  predicted  RSS 
classification  if  and  only  if  the  RSS  classification  codes  differed 
by  one  or  less.  For  example,  if  the  NSM  Expen  predicted  an 
RSS  classification  code  of  6  whereas  the  covariance  results 
determined  a  performance  of  Class  7,  a  match  was  declared.  A 
match  would  also  have  been  declared  if  the  actual  performance 
was  Class  5.  Since  the  NSM  Expen  is  only  expected  to  estimate 
a  hybrid’s  performance,  it  is  allowed  some  room  for  error. 

In  total,  the  NSM  Expen  System  was  run  four  hundred  and 
eighty-eight  times  in  order  to  determine  the  number  of  matches 
for  each  system  on  the  test  trajectories.  Figure  10  shows  the 
NSM  Expen's  performance  in  predicting  the  RSS  position  error 
for  each  hybrid  configuration.  The  predictive  performance 
metric  for  each  navaid  is  defined  as  the  percentage  of  number  of 
matches  obtained  from  the  total  number  of  combinations  tested 


for  that  navaid.  The  matches  on  all  four  trajectory  legs  are 
reflected  in  this  figure. 

The  NSM  Expen  performed  very  well  on  the  two  test 
trajectories.  Figure  10  shows  that  the  NSM  Expen  correctly 
predicts  the  RSS  position  error  better  than  70%  of  the  time  on 
test  Trajectory  #1.  The  system  required  only  the  trajectory 
information  and  its  knowledge  of  hybrid  system  performance  to 
make  these  predictions.  However,  its  pr^ictive  capability  on 
test  Trajectory  #2  is  slightly  worse  for  the  LORAN  hybrids, 
considerably  worse  for  the  VOR  and  Air  Data  sensor  hybrids, 
and  identical  for  the  remaining  configurations.  Hence,  the 
results  from  Trajectory  #2  suggest  that  additional  investigation 
into  trajectory  effects  on  VOR's  and  Air  Data  Sensor's 
performance  may  be  necessary. 


RADAR  S0«OR 

EXPERT  SYSTEMS 


The  results  in  Fig.  10  are  truly  encouraging  for  designers  of 
expert  systems.  We  have  shown  that  an  expert  system  can  be 
designed  from  data,  and  that  good  results  are  obtainable  even 
from  relatively  small  training  sets.  Recall  that  the  total  number 
of  examples  used  to  obtain  the  NSM  decision  trees  was  slightly 
less  than  one  thousand. 

CONCLUSIONS 

The  performances  of  seven  navigation  systems  aiding  a 
medium-accuracy  INS  were  investigated  using  Kalman  Filter 
covariance  analyses.  Hybrid  performance  decisions  were  based 
on  the  RSS  position  error  history  metric.  A  NSM  Expert  was 
designed  from  covariance  simulation  data  using  a  systematic 
method  comprised  of  the  two  statistical  techniques,  the  Analysis 
of  Variance  (ANOVA)  method  and  the  ID3  algorithm. 

ANOVA  results  show  that  statistically  different  position 
accuracies  are  obtained  when  different  navaids  are  used,  the 
number  of  radio  navigation  ground  stations  or  GPS  satellites 
used  to  aid  the  INS  is  varied,  the  aircraft's  trajectory  is  varied, 
and  the  performance  history  is  varied.  By  indicating  that  these 
four  factors  significantly  affect  the  decision  metric,  an 
appropriate  parameter  framework  was  designed,  and  a 
simulation  example  base  was  created. 


The  example  base  was  composed  of  over  nine  hundred 
training  examples  from  nearly  three  hundred  simulations.  The 
example  base  was  divided  into  seventeen  smaller  groups  to 
enhance  manageability.  The  ID3  algorithm  then  was  used  to 
determine  the  NSM  Expert's  classification  "rules"  in  the  form  of 
decision  trees.  The  performances  of  these  decision  trees  were 
assessed  on  two  arbitrary  trajectories,  by  counting  the  number 
of  times  the  rules  coirectly  predicted  the  RSS  position  accuracy. 
These  performance  results  then  were  presented  using  a 
predictive  metric. 

The  ANOVA/ID3  method  was  very  effective  for  the 
systematic  development  of  the  NSM  Expert  using  simulation 
data.  Results  show  that  the  NSM  Expen  can  predict  the  RSS 
position  accuracy  between  65  and  100%  of  the  time  for  a 
specified  navaid  configuration  and  aircraft  trajectory.  The  test 
trajectories  used  to  evaluate  the  sv  stem's  performance  show  that 
the  NSM  Expert  adapts  to  new  situations  and  provides 
reasonable  estimates  of  the  expected  hybrid  performance.  The 
system's  good  performance  with  relatively  few  examples  clearly 
shows  how  the  1D3  algorithm  maximizes  the  information 
content  contained  in  the  example  base.  The  performance  results 
strongly  suggest  that  operational  systems  can  be  designed  from 
simulation  or  experimental  data  using  the  ANOVA/ID3  method 
for  knowledge  acquisition.  The  systematic  nature  of  the  method 
makes  it  a  useful  tool  for  expert  system  designers. 

Other  aerospace  applications  that  are  good  candidates  for 
the  ANOVA/ID3  method  are  air  combat  pilot  strategies  from 
simulation  or  flight  test  data  and  air  traffic  control  solutions  to 
multi-configuration  problems.  The  expert  system  design 
methodology  also  is  pertinent  to  problems  such  as  nuclear 
reactor  control  strategies,  chemical  process  control  strategies, 
automated  highway  driving,  and  robotics  applications.  In  each 
case  simulation  or  operational  experiments  may  be  executed  for 
the  systematic  development  of  an  expen  system  advisor. 
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